Experiments in Automatic Benchmark Synthesis

In the nineteen-eighties, synthetic workloads such as Whetstone and Dhrystone fell out of favor as benchmarks of computer performance because they became unrepresentative of the performance of continuously-evolving applications. Hand-coded synthetic benchmarks take work to develop and maintain, are language feature specific, and are subject to compiler optimizations that eliminate code meant to make a significant contribution to performance. We present an automatic benchmark synthesis method that addresses these problems. The method automatically creates C-code that, when compiled and executed, is representative of the features of a target application but executes in a fraction of the original runtime. Our benchmark synthesis technique takes an actual executable, performs control flow analysis and workload characterization, and generates a representative synthetic benchmark. The representative sequences of instructions are instantiated as in-line assembly-language instructions in the synthetic benchmark. We synthesize versions of the SPEC95 and STREAM benchmarks with both perfect branching and a simple branching model. We find that benchmarks can be synthesized to an average IPC within 3.9% of the average IPC of the target benchmarks with remarkably similar instruction mix, cache access characteristics, RUU occupancies, and dependency characteristics. In addition, the change in IPC for a synthetic benchmark due to a design change is found to be proportional to the change in IPC for the original application. The synthesized versions of the SPEC95 benchmarks execute in 0.1% of the original execution time.